Improve plots and fix exploratory notebook issues by JohT · Pull Request #596 · JohT/code-graph-analysis-pipeline

JohT · 2026-05-16T15:37:22Z

🚀 Feature

⚙️ Optimization

🛠 Fix

Fix exploratory Jupyter notebooks

📖 Documentation

Copilot

Pull request overview

This PR makes a set of fixes and improvements to exploratory Jupyter notebooks and the related chart-generation Python scripts. It adds nbformat so Plotly can render figures in notebooks, repoints notebooks at their new queries/ folders (and ../../../cypher/... for shared Cypher), adds a new cyclomatic-complexity distribution chart + CSV report for the Java domain, drops the no-longer-needed openTSNE dimensionality reduction in favour of UMAP, and introduces a CI workflow that smoke-executes every explore/*.ipynb to catch import/syntax regressions.

Changes:

Notebook hygiene: type annotations, path corrections, removal of dead helpers (get_plotly_figure_write_image_settings, t-SNE path), stringification of date columns for Plotly JSON serialization, NEO4J_INITIAL_PASSWORD validation.
New chart pipeline: Cyclomatic_Method_Complexity_Distribution.cypher + CSV export + normalized per-artifact line chart in javaCharts.py; pie charts in externalDependencyCharts.py now plot the full dataset and rely on threshold filtering instead of pre-head(20).
New internal-check-notebooks.yml workflow that executes notebooks with nbconvert --allow-errors and fails only on ModuleNotFoundError/ImportError/SyntaxError.

Reviewed changes

Copilot reviewed 14 out of 15 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
pyproject.toml	Add `nbformat==5.10.4` (Plotly notebook rendering).
conda-environment.yml	Mirror the new `nbformat=5.10.4` pin.
uv.lock	Lockfile updates for `nbformat` and transitive deps.
domains/java/queries/method-metrics/Cyclomatic_Method_Complexity_Distribution.cypher	New query (description has a duplicated word; uses lowercase `asc`).
domains/java/javaCsv.sh	Export the new cyclomatic-complexity CSV.
domains/java/javaCharts.py	Refactor line-count chart to normalized line chart; add cyclomatic chart.
domains/java/explore/MethodMetricsJavaExploration.ipynb	Notebook fixes & path update for the moved cypher file.
domains/external-dependencies/externalDependencyCharts.py	Drop unnecessary `head(20)` calls; widen `<` to `<=` in drill-down filter (inconsistent with grouping).
domains/external-dependencies/explore/ExternalDependenciesJava.ipynb	Type annotations, password validation, path updates, bug-fix variable rename in drill-down cell.
domains/external-dependencies/explore/ExternalDependenciesTypescript.ipynb	Same modernization & path updates as the Java notebook.
domains/git-history/explore/GitHistoryGeneralExploration.ipynb	Cast Plotly date columns to strings; remove SVG write helper; metadata Python version downgraded (3.12.8) and diverges from conda env.
domains/node-embeddings/explore/NodeEmbeddingsJavaExploration.ipynb	Path updates and type-ignore pragmas.
domains/node-embeddings/explore/NodeEmbeddingsTypescriptExploration.ipynb	Same path/pragma updates.
domains/anomaly-detection/explore/NodeEmbeddingsHyperparameterTuningExploration.ipynb	Replace t-SNE with UMAP, add `copy=True` to HDBSCAN, cypher path updates; stray `execution_count: 43`.
.github/workflows/internal-check-notebooks.yml	New matrix workflow that smoke-tests all explore notebooks.

Copilot

Pull request overview

Copilot reviewed 14 out of 15 changed files in this pull request and generated 6 comments.

Copilot

Pull request overview

Copilot reviewed 16 out of 17 changed files in this pull request and generated 7 comments.

Comments suppressed due to low confidence (1)

domains/external-dependencies/externalDependencyCharts.py:535

Removing the top20 = ...head(20) pre-slicing here (and in all the other save_pie_chart_pair callsites below in this function and in generate_typescript_charts) means save_pie_chart_pair now receives the full overall/spread datasets, which can contain hundreds of rows. The chart relies entirely on group_small_values_into_others's percentage threshold to keep the pie readable. Please verify on a large project (e.g. AxonFramework) that the resulting pie charts and legends remain legible — without the explicit top-20 cap, charts with many low-but-not-tiny-percentage slices can become unreadable. If acceptable visually, consider documenting that the "Top N" naming in chart_name_prefix (Java_Top_external_packages_by_types, etc.) no longer reflects a strict top-N selection.


    # ── Top external packages (Table 1 equivalent) ────────────────────────────
    if not overall_data.empty:
        save_pie_chart_pair(
            source_data=overall_data,
            value_column="numberOfExternalCallerTypes",
            name_column="externalPackageName",
            chart_name_prefix="Java_Top_external_packages_by_types",
            primary_threshold_percent=0.7,
            report_directory=report_directory,
            verbose=verbose,
        )
        save_pie_chart_pair(
            source_data=overall_data,
            value_column="numberOfExternalCallerPackages",
            name_column="externalPackageName",
            chart_name_prefix="Java_Top_external_packages_by_packages",
            primary_threshold_percent=0.7,
            report_directory=report_directory,
            verbose=verbose,
        )

    # ── Second-level package grouping (Table 2 equivalent) ────────────────────
    if not second_level_overall_data.empty:
        save_pie_chart_pair(
            source_data=second_level_overall_data,
            value_column="numberOfExternalCallerTypes",
            name_column="externalSecondLevelPackageName",
            chart_name_prefix="Java_Top_second_level_packages_by_types",
            primary_threshold_percent=0.7,
            report_directory=report_directory,
            verbose=verbose,
        )
        save_pie_chart_pair(
            source_data=second_level_overall_data,
            value_column="numberOfExternalCallerPackages",
            name_column="externalSecondLevelPackageName",
            chart_name_prefix="Java_Top_second_level_packages_by_packages",
            primary_threshold_percent=0.7,
            report_directory=report_directory,
            verbose=verbose,
        )

    # ── Most spread external packages (Table 3 equivalent) ────────────────────
    if not spread_data.empty:
        save_pie_chart_pair(
            source_data=spread_data,
            value_column="sumNumberOfTypes",
            name_column="externalPackageName",
            chart_name_prefix="Java_Most_spread_packages_by_types",
            primary_threshold_percent=0.5,
            report_directory=report_directory,
            verbose=verbose,
        )
        save_pie_chart_pair(
            source_data=spread_data,
            value_column="sumNumberOfPackages",
            name_column="externalPackageName",
            chart_name_prefix="Java_Most_spread_packages_by_packages",
            primary_threshold_percent=0.5,
            report_directory=report_directory,
            verbose=verbose,
        )

    # ── Most spread second-level packages (Table 4 equivalent) ────────────────
    if not second_level_spread_data.empty:
        save_pie_chart_pair(
            source_data=second_level_spread_data,
            value_column="sumNumberOfTypes",
            name_column="externalSecondLevelPackageName",
            chart_name_prefix="Java_Most_spread_second_level_packages_by_types",
            primary_threshold_percent=0.5,
            report_directory=report_directory,
            verbose=verbose,
        )
        save_pie_chart_pair(
            source_data=second_level_spread_data,
            value_column="sumNumberOfPackages",
            name_column="externalSecondLevelPackageName",
            chart_name_prefix="Java_Most_spread_second_level_packages_by_packages",
            primary_threshold_percent=0.5,
            report_directory=report_directory,

JohT changed the title ~~Feature/improve plots~~ Improve plots and fix exploratory notebook issues May 16, 2026

JohT self-assigned this May 16, 2026

JohT force-pushed the feature/improve-plots branch from e8d33a3 to 6705027 Compare May 16, 2026 18:41

JohT requested a review from Copilot May 16, 2026 18:41

Copilot started reviewing on behalf of JohT May 16, 2026 18:43 View session

Copilot AI reviewed May 16, 2026

View reviewed changes

JohT force-pushed the feature/improve-plots branch from 6705027 to ddf3dea Compare May 16, 2026 18:54

JohT requested a review from Copilot May 16, 2026 19:36

Copilot started reviewing on behalf of JohT May 16, 2026 19:36 View session

Copilot AI reviewed May 16, 2026

View reviewed changes

JohT marked this pull request as ready for review May 16, 2026 20:05

JohT force-pushed the feature/improve-plots branch 3 times, most recently from 5a0913f to 5a0c57f Compare May 17, 2026 06:31

JohT requested a review from Copilot May 17, 2026 08:23

Copilot started reviewing on behalf of JohT May 17, 2026 08:24 View session

Copilot AI reviewed May 17, 2026

View reviewed changes

JohT added 8 commits May 17, 2026 11:50

Improve charts to look similar to original explore notebooks

f2148f2

Fix exploratory Jupyter notebooks

87f679c

Add Python lib nbformat for plotly in Juypter notebooks

3b7ed50

Remove setuptools dependency since opentsne is removed

4d0badc

Add pipeline to quickly validate Jupyter notebooks

047a93d

Prepare release v4.0.1

afa6b70

Fix broken links in README.md

77b6958

Improved domains overview in documentation

9347997

JohT force-pushed the feature/improve-plots branch from 346969a to 9347997 Compare May 17, 2026 09:50

JohT merged commit e7db08c into main May 17, 2026
14 checks passed

JohT deleted the feature/improve-plots branch May 17, 2026 10:33

Conversation

JohT commented May 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🚀 Feature

⚙️ Optimization

🛠 Fix

📖 Documentation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

JohT commented May 16, 2026 •

edited

Loading